System and Methods for Calibration of an Array Camera

ABSTRACT

Systems and methods for calibrating an array camera are disclosed. Systems and methods for calibrating an array camera in accordance with embodiments of this invention include the detecting of defects in the imaging components of the array camera and determining whether the detected defects may be tolerated by image processing algorithms. The calibration process also determines translation information between imaging components in the array camera for use in merging the image data from the various imaging components during image processing. Furthermore, the calibration process may include a process to improve photometric uniformity in the imaging components.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. application Ser. No.14/203,308 entitled “System and Methods for Calibration of an ArrayCamera”, filed Mar. 10, 2014, which application claims priority to U.S.Provisional Patent Application No. 61/780,748 entitled “System andMethods for Calibration of an Array Camera,” filed Mar. 13, 2013, thedisclosures of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to calibration of an array camera. Moreparticularly, this invention relates to systems and methods forcalibrating an array camera to detect defects in individual imagingcomponents of the array camera and to gather information that can beutilized to perform image processing with image data from the imagingcomponents in an array camera.

BACKGROUND

An array camera is a camera that is made up of multiple imagingcomponents. Each individual imaging component captures data for a twodimensional image of a view. For purposes of this discussion, an imagingcomponent is an individual camera and/or circuitry including an array ofpixels that capture data for a two dimensional image of the view. Thedata of the two-dimensional images can be used to producesuper-resolution (SR) images and other types of images of the view usingsome or all of the data from the various two dimensional images capturedby the individual imaging components.

During the manufacture of most digital cameras (including array cameras)and each imaging component of an array camera, a calibration process istypically performed. In a conventional camera or imaging component, thecalibration process typically measures a number of characteristics ofthe imaging component that must be quantified to ensure effective imageprocessing. The calibration of an array camera requires specializedcalibration processes to quantify certain characteristics unique to anarray camera and not particularly common in conventional digital camerasor imaging components, such as, the uniformity of various intrinsiccharacteristics among the elements of the array and the extrinsicgeometric relationships between the elements of the array.

One aspect of an array camera that is different from conventionalcameras is that some types of defects in pixels of an individual imagingcomponent may be compensated for during image processing by usinginformation collected from another imaging component in the array. Thus,an array camera is more tolerant of defects in the individual imagingcomponents.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of this inventionenable calibration of array cameras. In accordance with embodiments ofthis invention, one or more of the imaging components of the arraycamera are designated as a reference imaging component and each of theremaining imaging components in the array camera is an associate imagingcomponent. Each of the associate imaging components is associated withat least one of the reference imaging components. In accordance withembodiments of the invention, the calibration of an array cameraincludes performing a defect detection process and a geometriccalibration process. The defect detection process identifies pixeldefects in each of the imaging components in the array camera anddetermines whether the identified pixel defects in a particular imagingcomponent may be corrected in image processing using image data from atleast one other one of the plurality of imaging components. Thegeometric calibration process determines translation information thatrelates image data from particular pixels in a reference imagingcomponent to image data from corresponding pixels in each associateimaging component associated with the reference imaging component.

In accordance with some embodiments of the invention, the calibrationprocess also includes a Modulation Transfer Function (MTF) calibrationprocess that is performed on each of the imaging components in the arraycamera to determine MTF data over the Field of View (FoV) or the FieldHeight (FH) for each of the imaging components. The MTF calibrationprocess includes capturing image data of a target with each of theimaging components and analyzing the image data captured by each ofimaging components to determine MTF data over one of a FoV and FH foreach of the imaging components in accordance with a number ofembodiments. The MTF data for each of the imaging components isdetermined from the MTF detected in the image data and is stored inmemory.

In accordance with many embodiments of the invention, the calibrationprocess also includes a photometric calibration process performed foreach of the plurality of imaging components in the array camera. Thephotometric calibration process determining image data adjustments foreach of the imaging components to provide image data from each of theimaging components with a uniform intensity profile. In accordance witha number of embodiments, the photometric calibration is performed in thefollowing manner. Image data of a flat field target under controlledlighting conditions is captured with each of the imaging components inthe array camera. The captured image data from each of the imagingcomponents is analyzed to determine a per pixel adjustment needed tomatch coŝ4 fall-off response for an image and a grid of adjustments isgenerated for each of the imaging components. Each adjustment in thegrid indicates the adjustment information for data from a set of pixelsin a particular imaging component needed to match the coŝ4 fall-offresponse for an image.

In accordance with some embodiments of the invention, the defects ineach of the imaging components are identified in the following manner. Afirst set of image data of a field target under high intensity lightingis captured with each of the imaging components. A second set of imagedata of a field target under low intensity lighting is also capturedwith each of the imaging components. For each imaging component, thefirst and second sets of image data are analyzed to identify each pixelhaving an unexpected response value. The identified pixels are flaggedwith a category indicating the category unexpected response received andthe identified pixels and flagged categories of the pixels are stored.In accordance with a number of embodiments, the detecting of defects ineach imaging component further determining the number of each categoryof defects identified in each imaging component, comparing the number ofeach category of defects in each imaging component to a threshold, andgenerating a disposition output for a particular imaging component inresponse to the number of defects in a category for the particularimaging component being greater than the threshold.

In accordance with some embodiments, the determining of whether a defectmay be corrected during image processing is performed in the followingmanner. A defect a first imaging component in the array camera isselected and a parallax translation path for the defect over a range ofsupported distances is determined. A parallax translation path over therange of supported distances from each defect in other imagingcomponents of the array camera is also determined. A determination ismade as to whether a predetermined number of parallax translation pathsof defects in other imaging components intersect the parallaxtranslation path of the selected defect. A disposition output for thefirst imaging component is generated if a predetermined number ofparallax translation paths for defects from other images componentsintersect the parallax translation path of the selected defect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 conceptually illustrates an array camera in accordance with anembodiment of this invention.

FIG. 2 illustrates a flow diagram of a calibration process for an arraycamera in accordance with embodiments of this invention.

FIG. 3 illustrates a flow diagram of a defect detection processperformed during a calibration process of an array camera in accordancewith embodiments of this invention.

FIG. 4 illustrates a flow diagram of a defect tolerance determinationprocess performed during a calibration of an array camera to determinewhether certain defects in imaging components of the array camera may behandled during image processing in accordance with embodiments of thisinvention.

FIG. 5 illustrates a flow diagram of a photometric calibration processfor imaging components performed during a calibration process for anarray camera in accordance with embodiments of this invention.

FIG. 6 illustrates a flow diagram of a geometric calibration processperformed during a calibration process to gather information relatingpixel information of associate and reference imaging components in thearray camera in accordance with embodiments of the invention.

FIG. 7 illustrates a flow diagram of an MTF calibration processperformed during a calibration process to obtain MTF information foreach imaging component of an array camera in accordance with embodimentsof this invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for performing acalibration process for an array camera to detect potential defects inthe imaging components and to gather information that can be utilizedduring image processing in accordance with embodiments of the inventionare illustrated. In accordance with many embodiments of the invention,at least one of the imaging components in the array camera is designatedas a “master” or “reference” imaging component; and each of theremaining imaging components in the array is an associate imagingcomponent. Each associate imaging component is associated with at leastone of the reference imaging components. During use, the referenceimaging component can be utilized to capture a reference image andassociate imaging components can capture alternate view images that canbe utilized in combination with the reference image to synthesize ahigher resolution image from the viewpoint of the reference imagingcomponent. In several embodiments, a calibration process is performed todetermine relevant information about each imaging component including(but not limited to) colorimetric, MTF, and photometric information. Acalibration process may also determine geometric information thatrelates image data collected by the imaging components to the image datacollected by other imaging components to assist in integrating the imagedata during image processing. The calibration process may also identifydefects in particular imaging arrays and gather information that may beused to correct the identified defects during image processing.

Capturing Light Field Image Data

A light field, which is often defined as a 4D function characterizingthe light from all direction at all points in a scene, can beinterpreted as a two-dimensional (2D) collection of data from 2D imagesof a scene. Array cameras, such as those described in U.S. patentapplication Ser. No. 12/935,504 entitled “Capturing and Processing ofImages using Monolithic Array camera with Heterogeneous Imagers” toVenkataraman et al., can be utilized to capture light field images. In anumber of embodiments, super-resolution processes such as thosedescribed in U.S. patent application Ser. No. 12/967,807 entitled“Systems and Methods for Synthesizing High Resolution Images UsingSuper-Resolution Processes” to Lelescu et al., are utilized tosynthesize a higher resolution 2D image or a stereo pair of higherresolution 2D images from the lower resolution image data in the lightfield captured by the imaging components in an array camera. The termshigh or higher resolution and low or lower resolution are used here in arelative sense and not to indicate the specific resolutions of theimages captured by the array camera. The super-resolution processingtechniques described in U.S. patent application Ser. No. 12/967,807 canutilize a variety of calibration information including scene independentgeometric corrections related to the distortions introduced by theconstruction of the imaging components utilized to capture the lowerresolution image data. Scene independent distortions introduced by theimaging components of an array camera can limit the fidelity of a higherresolution image generated by super-resolution imaging due touncertainty concerning the relevant location within the scene of a pieceof image data. By measuring the distortions during calibration, theuncertainty introduced by the distortions can be reduced, improving theperformance of the super-resolution process. In a similar manner,colorimetric, MTF, and photometric variations between similar imagingcomponents in a array camera can complicate image processing byincreasing the difficulty of identifying similar points within a scenein the image data captured by the imaging components. The disclosures ofU.S. patent application Ser. No. 12/935,504 and U.S. patent applicationSer. No. 12/967,807 are hereby incorporated by reference in theirentirety.

Each two-dimensional (2D) image in a captured light field is from theviewpoint of one of the imaging components in the array camera. A highresolution image synthesized using super-resolution processing issynthesized from a specific viewpoint that can be referred to as areference viewpoint. The reference viewpoint can be from the viewpointof one of the imaging components in an array camera. Alternatively, thereference viewpoint can be an arbitrary virtual viewpoint.

Due to the different viewpoint of each of the imaging components,parallax results in variations in the position of foreground objectswithin the image data of the scene from the various imaging componentsof the array camera. Processes for performing parallax detection arediscussed in U.S. Provisional Patent Application Ser. No. 61/691,666entitled “Systems and Methods for Parallax Detection and Correction inImages Captured Using Array Cameras” to Venkataraman et al., thedisclosure of which is incorporated by reference herein in its entirety.As is disclosed in U.S. Provisional Patent Application Ser. No.61/691,666, a depth map from a reference viewpoint can be generated bydetermining the disparity between the pixels in the images within alight field due to parallax. A depth map indicates the distance of thesurfaces of scene objects from a reference viewpoint. In a number ofembodiments, the computational complexity of generating depth maps isreduced by generating an initial low resolution depth map and thenincreasing the resolution of the depth map in regions where additionaldepth information is desirable such as (but not limited to) regionsinvolving depth transitions and/or regions containing pixels that areoccluded in one or more images within the light field.

During super-resolution processing, a depth map can be utilized in avariety of ways. U.S. patent application Ser. No. 12/967,807 describeshow a depth map can be utilized during super-resolution processing todynamically refocus a synthesized image to blur the synthesized image tomake portions of the scene that do not lie on the focal plane to appearout of focus. U.S. patent application Ser. No. 12/967,807 also describeshow a depth map can be utilized during super-resolution processing togenerate a stereo pair of higher resolution images for use in 3Dapplications. A depth map can also be utilized to synthesize a highresolution image from one or more virtual viewpoints. In this way, arendering device can simulate motion parallax and dolly zoom. Inaddition to utilizing a depth map during super-resolution processing, adepth map can be utilized in a variety of post processing processes toachieve effects including (but not limited to) dynamic refocusing,generation of stereo pairs, and generation of virtual viewpoints withoutperforming super-resolution processing.

Array Camera Architecture

Array cameras in accordance with embodiments of the invention areconfigured so that the array camera software can control the capture oflight field image data and can capture the light field image data into afile that can be used to render one or more images on any of a varietyof appropriately configured rendering devices. An array camera includingan array of imaging components in accordance with an embodiment of theinvention is illustrated in FIG. 1. The array camera 100 includes anarray camera module 102, a processor 108, a memory 110 and a display112. The array camera module 102 includes an array of imaging components104 formed by a sensor and a lens stack array and the array cameramodule 102 is configured to communicate with a processor 108. Theprocessor 108 is also configured to communicate with one or moredifferent types of memory 110 that can be utilized to store image dataand/or contain machine readable instructions utilized to configure theprocessor to perform processes including (but not limited to) thevarious processes described below. The processor 108 may also beconnected to a network via a network connection 106 to communicate withother processing systems connected to the network. The display 112 canbe utilized by the processor 108 to present a user interface to a userand to display an image rendered using the light field image data.Although the processor is illustrated as a single processor, arraycameras in accordance with embodiments of the invention can utilize asingle processor or multiple processors including (but not limited to) agraphics processing unit (GPU).

In the illustrated embodiment, the processor 108 receives image datagenerated by the array camera module 102 and reconstructs the lightfield captured by the array camera module 102 from the image data.Sensors including multiple focal planes that can be utilized in theconstruction of array camera modules are discussed in U.S. patentapplication Ser. No. 13/106,797 entitled “Architectures for System onChip Array Cameras”, to Pain et al., the disclosure of which isincorporated herein by reference in its entirety. The processor 108 canmanipulate the light field in any of a variety of different waysincluding (but not limited to) determining the depth and visibility ofthe pixels in the light field and synthesizing higher resolution 2Dimages from the image data of the light field.

In the illustrated embodiment, the imaging components 104 are configuredin a 5×5 array. Each imaging component 104 in the array camera module102 is capable of capturing an image of the view. The sensor elementsutilized in the imaging components 104 can be individual light sensingelements such as, but not limited to, traditional CIS (CMOS ImageSensor) pixels, CCD (charge-coupled device) pixels, high dynamic rangesensor elements, multispectral sensor elements and/or any otherstructure configured to generate an electrical signal indicative oflight incident on the structure. In many embodiments, the sensorelements of the imaging components 104 have similar physical propertiesand receive light via the same optical channel and color filter (wherepresent). In other embodiments, the sensor elements of the imagingcomponents 104 have different characteristics and, in many instances,the characteristics of the sensor elements are related to the colorfilter applied to each sensor element. In several embodiments, thesensor elements of an imaging component includes a plurality of rows ofpixels that also form a plurality of columns of pixels and the pixels ofeach imaging component are contained within a region of the sensor thatdoes not contain pixels from another imaging component.

In many embodiments, an array of images (i.e. a light field) is createdusing the image data captured by the imaging components 104. As notedabove, the processor 108 in accordance with many embodiments of theinvention is configured using appropriate software to take the imagedata within the light field and synthesize one or more high resolutionimages. In several embodiments, the high resolution image is synthesizedfrom a reference viewpoint, typically the viewpoint of a referenceimaging component 104 within the array camera module 102. In manyembodiments, the processor is able to synthesize an image from a virtualviewpoint, which does not correspond to the viewpoints of any of theimaging components 104 in the array camera module 102. Unless all of theobjects within a captured scene are a significant distance from thearray camera, the images in the light field will include disparity dueto the different fields of view of the focal planes used to capture theimages. Processes for detecting and correcting for disparity whenperforming super-resolution processing in accordance with embodiments ofthe invention are discussed in U.S. Provisional Patent Application Ser.No. 61/691,666 (incorporated by reference above). The detected disparitycan be utilized to generate a depth map. The high resolution image anddepth map can be encoded and stored in memory 110 in a light field imagefile. The processor 108 can use the light field image file to render oneor more high resolution images. The processor 108 can also coordinatethe sharing of the light field image file with other devices (e.g. via anetwork connection), which can use the light field image file to renderone or more high resolution images.

Although a specific array camera architecture is illustrated in FIG. 1,systems and methods for performing calibration of an array camera inaccordance with embodiments of the invention may be performed on anyarray camera configuration appropriate to the requirements of a specificapplication. Systems and methods for performing a calibration processfor an array camera in accordance with embodiments of the invention arediscussed below.

Calibration Process for an Array Camera

An array camera has features beyond those in a conventional camera thatare characterized during a calibration process to assistsuper-resolution processing algorithms to produce images of a desiredresolution. During a manufacture process and/or periodically during thelife of an array camera, a calibration process may be performed togenerate or update information that can be utilized to perform thesuper-resolution processing algorithms. A calibration process to collectthe calibration information in accordance with embodiments of thisinvention is illustrated in FIG. 2.

A Calibration process (200) can involve performing a defect detectionprocess (205), a photometric calibration process (210), a geometriccalibration process (215), and/or a MTF calibration process (220). Oneskilled in the art will recognize that a calibration process may includeother processes for detecting other types of defects and/or to collectinformation about other aspects of the array camera and the underlyingimaging components if other types of information are needed to performthe image processing algorithms without departing from this invention.The defect detection process (205) is performed to identify pixels ineach of the imaging components that have defects and to determine howinformation from a defective pixel is to be handled during imageprocessing. A defect detection process in accordance with someembodiments of this invention is described below with reference to FIGS.3 and 4. The photometric calibration process (210) determinesinformation used during image processing to adjust the data of a viewcaptured by each of the imaging components so that the data of the viewreceived from each of the imaging components has a uniform intensityprofile. A photometric process performed in accordance with someembodiments of this invention is discussed below with reference to FIG.5. The geometric calibration process (215) determines the informationused to perform a translation to align image data from pixels in eachassociate imaging component with image data from corresponding pixels ina reference imaging component. A geometric calibration process performedin accordance with some embodiments of this invention is described belowwith reference to FIG. 6. The MTF calibration process (220) is performedto characterize the “sharpness” of the optics over the FoV of eachimaging component in order to adjust the data from each of the imagingcomponents to cause the image data from the various imaging componentsto have an essentially uniform MTF. A MTF calibration process performedin accordance with some embodiments of this invention is described belowwith reference to FIG. 7.

Furthermore, calibration process 200 and/or each of the individualcalibration/error detection processes included in process 200 may beperformed one or more times at each of several predeterminedtemperatures and the results may be aggregated into calibrationinformation that is indexed by temperature in accordance with a numberof embodiments of this invention. In accordance with these embodiments,the temperature of the environment and/or the array camera being testedat the time of an iteration of the process 200 is optionally determined(202). The temperature may be measured by a connected device such as,but not limited to, a dark sensor or may be input by an operatordepending on the embodiment. The determined temperature is associatedwith the information generated during the iteration. Furthermore,process 200 may determine whether additional iterations need to beperformed at other temperatures (225). If so, process 200 is repeatedafter the temperature of a test environment is adjusted. Otherwiseprocess 200 ends.

Defect Detection

A defect detection process is a process that detects defects inindividual pixel sensors of the imaging components in the array cameraand then determines how data from the identified defective pixel sensorsis to be handled. Like conventional digital imagers, the imagingcomponents in an array camera often have typical pixel sensor defectsincluding, but not limited to, cold pixels, cool pixels, warm pixels,hot pixels, column failures, row failures, non-uniformities, and randomtelegraph pixel failures. However, conventional digital camerastypically have set limits as to the number of defects in each class thatcan be tolerated in an imager. These limits are often zero or near zerowhen the failures are in a critical area of the pixel array (i.e. in thecenter of the pixel array). An array camera, on the other hand, may havea higher tolerance to these defects in individual imaging componentsbecause an image from an array camera is synthesized from image datareceived from one or more imaging components. Thus, image processingalgorithms may be able to use image data from another imaging componentto provide data about the image in the area covered by the image datafrom the defective pixel sensor in one of the imaging components.Because of this, a calibration process for an array camera shouldidentify the defective pixels in each of the imaging components anddetermine if the identified defects can be tolerated during imageprocessing. A process for identifying pixel defects in each of theimaging components of an array camera in accordance with an embodimentof the invention is illustrated in FIG. 3 and a process for determiningwhether certain types of defects in a particular imaging component maybe tolerated during image processing in accordance with some embodimentsof this invention is illustrated in FIG. 4.

To identify pixel defects, process (300) captures at least two images ofa flat field target with the array camera under controlled lighting. Thecapturing of the image with the array camera captures image data foreach of the imaging components in the array. A first one of the imagesis captured under high intensity lighting (305) and a second one of theimages is captured under low intensity lighting (310). The detection ofdefective pixels may be performed in the following manner based on imagedata from the images captured. An imaging component is selected (315).The image data from the high intensity image (305) and low intensityimage (310) captured by the selected imaging component is analyzed(320). The analyzing may include, but is not limited to, determiningwhether the value of a particular pixel is in agreement with the valuesof adjacent pixels or within an acceptable standard deviation from thevalues of the adjacent pixels; determining the value of the pixel iswithin a predetermined range or with an acceptable deviation from therange; and/or detecting patterns of values within an array of pixelsthat are indicative of a defect. From the analysis, pixels having anunexpected response value are identified (325). The following areexamples of the categories of unexpected values that may be flagged inthe identified pixels:

TYPE OF DEFECT DEFECT VALUE Cold Pixel values below a Cold PixelThreshold, such as 10 Hot Pixel values above a Hot Pixel Threshold, suchas 245 Cool Pixel values below the Cool Pixel Threshold from a nominalneighborhood value Warm Pixel values above a Warm Pixel Threshold fromthe nominal neighborhood value Cluster Fault values outside the ClusterFault threshold from Threshold the nominal neighborhood value A2D faultone or value bits stuck at “0” or “1” Row fault special type of clusterfault Column Fault special type of cluster fault

The identified pixels are stored and are flagged into categories (330)based on the type of unexpected response values received. The defectivepixel location and indication of the category defect are stored forfuture use by the imaging processing of the array camera. The flaggingand storing of the defects may be performed by generating a sorted listof identified defective pixels including a value indicating the type ofdefect of the pixel, such as the values shown above. The list is thensearched for adjacent pixels having similar error values to identifyclusters. In accordance with some embodiments, the search may beperformed using two nested loops with two escape clauses. The defectvalues for the identified pixels in the detected cluster faults are thenchanged to reflect the appropriate cluster faults.

After all of the defects have been identified, the number of identifiedpixels having each particular type of defect is calculated and stored(335). The list of defective pixels may also be stored as defect dataand any number of different data techniques may be used to reduce thesize of the stored defect data. The number of each type of defect isthen compared to a threshold. The threshold may be a failure limit for aparticular type of error that may be determined by a manufacturer ordesigner. If the number of a particular type is greater than thethreshold for the type (340), the camera is set for a particulardisposition (345). For example, the camera may be set to “fail”, “failwith too many defects”, and/or “fail with logic faults”. The dispositionmay then be used in the manufacturing process to correct the defects inthe particular imaging components or determine further processing stepsin the manufacturing process for the array camera.

However, as discussed above, the image processing algorithms used in anarray camera may be used to compensate for some types of defects in aparticular imaging component. Thus, a tolerance test is performed forthe identified defects in the selected imaging component (350). Atolerance test in accordance with some embodiments of this invention isdiscussed below with reference to FIG. 4. Process (300) is then repeatedfor each of the remaining imaging components until the images for all ofthe imaging components have been analyzed (355).

Unlike conventional digital camera, certain pixel defects in individualimaging components including, but not limited to, cluster defects may betolerated in an array camera because other imaging components mayprovide the same information as the defective pixel and or cluster ofpixels. A process for determining whether particular defects in animaging component may be tolerated during the image processing in thearray camera in accordance with embodiments of this invention isillustrated in FIG. 4. An identified defect is selected (405) from theimaging component being analyzed. Typically, the defects that areselected are limited to certain types of defects. For example, thedefects that may be selected may be limited to cluster defects having asize less than or greater than a predetermined threshold.

Another imaging component is also selected (410). An identified defectfrom the selected imaging component is then chosen (415). The processthen determines whether the selected defect (415) is within aprogrammable distance from the selected defect (405) from the imagingcomponent under consideration. If the defects (405) and (415) from thedifferent imaging components are within a programmable distance, theprocess determines a parallax translation path over a range of supportedimaging distances for each of the defects (420). The paths are comparedto determine whether the paths intersect (425). If the paths intersect,the process stores the intersection and distance (430). A defectintersection counter is then incremented and stored (435).

The defect intersection counter is then compared to a threshold (440).If the defect intersection counter is greater than the threshold, thecamera is set for a particular disposition (445). This is to determinewhether there is image data for the area of the image covered by theimage data from the defect under consideration to still generate imageinformation for that area. Alternatively, if there is not enough imagedata for the area, the camera is set to a particular disposition tohandle the defect through conventional means. The process continues(450) until all the defects in the selected imaging component (410) areanalyzed. After all defects in the selected imaging component (410) arecompared, the process repeats (455) with respect to each of the otherimaging components in the array camera. After the selected defect (405)has been compared to the defects in all of the other imaging components,the process is repeated (460) from (405) for each of the otherselectable defects of the imaging component under consideration.Although specific processes for performing defect detection and forassessing the significance of defects are described above, any of avariety of processes can be utilized to detect defects and to determinewhether patterns of defects are likely to impact images synthesizedusing image data captured by an array camera in accordance withembodiments of the invention.

Photometric Calibration

In accordance with many embodiments of this invention, a photometriccalibration process is performed on the array camera. The photometriccalibration is performed to adjust the image data captured by thedifferent imaging components to have a uniform intensity profile so thatcorresponding pixels in the different imaging components havesubstantially equal values modulo the spectral band being imaged whenimaging the same object in physical space. A photometric calibration inaccordance with an embodiment of this invention is illustrated in FIG.5.

Photometric calibration process (500) captures an image of a flat fieldtarget with the array camera (505). The capturing of the image with thearray camera generates image data of the flat field target for each ofthe imaging components in the array camera. The process then does aphotometric calibration process for each of the imaging components.

The photometric process for imaging components begins by retrieving theimage data for a selected imaging component (510). The image data forthe selected imaging component is analyzed (515). The analysis isperformed by viewing the image data as an intensity surface. If theimage data were ideal, the image data would have a coŝ4 fall-offresponse across the Field of View. The image data is analyzed todetermine the per-pixel adjustment that is needed to change theintensity of each pixel to match a profile corresponding to the coŝ4fall-off that would be associated with a pixel that varies from zero (0)at the optical center to some value, typically about 15 percent less, ata predetermined Field Height (520). This may be performed by detecting ahorizontal line in the image data and performing line filtering on thedetected horizontal line, and fitting a coŝ4 profile to the data. Thisis repeated with horizontal lines throughout the image data. In otherembodiments, any of a variety of line fitting techniques can be utilizedas appropriate to the requirements of specific applications.

As it is common that the variation across the Field of View of anyimaging device is low frequency, the process may generate a gridrepresenting adjustments of the data from the pixels in a step-sizemanner (525). The grid includes the multipliers that translate thefitted curves to the desired curves from line data in a particular areaof the image data. The grid data is then stored in an appropriate fileor data structure for the particular imaging component (530-535). Thegrid data in the file or data structure is then used in the processingpipeline to adjust the intensity profiles in the image data receivedfrom the particular imaging component. The process (500) is thenrepeated (540) for each of the remaining imaging components in the arraycamera. In accordance with a number of embodiments, a photometriccalibration process may include steps that balance the response betweenimaging components in the same spectral bands; adjust a maximum, aminimum, average, and/or other parameter of a group; and other stepsthat may be needed to compensate for errors that affect the photometricsof an array camera.

Although specific processes for performing photometric calibration aredescribed above with reference to FIG. 5, any of a variety of processescan be utilized to determine photometric calibration data for use in thephotometric calibration of image data captured by array cameras inaccordance with embodiments of the invention. Processes for performinggeometric calibration of array cameras in accordance with embodiments ofthe invention are discussed further below.

Geometric Calibration

During the calibration process of an array camera, the translationinformation relating a position of a pixel in a reference imagingcomponent to a position of a corresponding pixel in an associatedimaging component can be determined in order to assist in fusing mergethe image data captured by the various imaging components. This processis referred to as geometric calibration of the imaging components. Aprocess for performing a geometric calibration of the reference andassociate imaging components of an array camera in accordance with anembodiment of this invention is illustrated in FIG. 6.

In process (600), an image of a single target is captured with an arraycamera (605). The capturing of the image with the array camera involvescapturing image data of the single target from each of the imagingcomponents in the array camera. The single target image may include aset of easily identified control points. In accordance with someembodiments, the target may include a low contrast slanted edge patternsuch as a checkerboard pattern. The target should change with a highenough frequency to provide a sufficient number of control points toidentify an expected level of low-frequency non-uniformities in theoptical systems of the imaging components.

For each reference imaging component, the following process may be usedto determination the translation information that relates the associateimaging components associated with the reference imaging component. Theimage data from for a reference imaging component is retrieved (610).The location of the control points in the image data of the referenceimaging component are identified (615). The locations of control pointsare identified to a reasonably fine sub-pixel level, typically to within0.1 pixel accuracy.

In accordance with some embodiments, distortion information for thecaptured image optionally is measured (617). The distortion informationis then used to modify control point information in the image data tocompensate for the measured distortion. The modified control pointinformation may then be used to correct geometric errors in theassociate imaging components that are associated with the referenceimaging component

The image data for an associate imaging component is then retrieved(620) and the locations of the control points in the image data of theassociate imaging component are determined (625) in a similar manner asdescribed above with respect to the reference imaging component (615).In accordance with several embodiments, the distortion in the image dataof the associate imaging component is optionally measured (627) in amanner similar to the measurement of the distortion in the image data ofthe reference imaging component. After the location of control pointsare identified in the reference image data and the associate image data,the displacements between corresponding points in the reference andassociate image data are determined (630). Adjustments may optionally bemade to the image data of the reference imaging component to performcertain geometric adjustments to the reference imaging component thatnormally would be performed during the image processing if not correctedby the translation between the reference and associate imagingcomponents. Such adjustments may be preferable where the basicprescription for a particular imaging device exhibits geometricaberrations such as pincushion and barrel distortion exceeding certainthreshold such as 10%. However, this threshold may be as low as 2-3% oreven lower in accordance with some embodiments. The displacement data isalso adjusted to account for the parallax shift associated with thedistance from the array camera to the single target (632). Furthermore,the displacement data may optionally be adjusted to correct forgeometric distortions measured in order to increase alignment accuracybetween the image data of the reference and associate imaging componentsin accordance with a number of embodiments.

The displacement information will typically be represented by irregulargrid displacement values across the image for the associate imagingcomponent. This may then be used to generate a regular grid ofdisplacement information (640). The grid may be adjustable in astep-size manner to account for frequency variations in the translationinformation across the image. The translation information in the gridmay then be used to generate a fusion data structure (645). The fusiondata structure includes a list of offsets along an x axis and a y axisand may be used to merge the image data from all of the image componentsinto a single image structure. The fusion structure essentially mapseach pixel of an associate imaging component into an appropriatephysical relation to the corresponding pixel in the reference imagingcomponent.

The translation information in the grid may also be used to generate asampling data structure (650). The sampling data structure includesinformation that allows the image data from an associate imagingcomponent to be sampled to obtain information that corresponds preciselyto the pixel sample points in the reference camera. In otherembodiments, any of a variety of data structures and/or techniques canbe utilized to store the translation information as appropriate to therequirements of specific applications.

The above described process (620-650) is then repeated for eachassociate imaging component that is associated with a particularreference component (655). The reference imaging component process(610-655) is then repeated for each reference imaging component in thearray camera to obtain the translation information for each associateimaging component associated with the reference imaging component untilall the grouping of reference and associate imaging components areprocessed.

Since the imaging components in the array camera observe a calibrationtarget from different points in space, there is some parallax shift inthe images from each of the imaging components. This parallax shift canbe calculated from measurements associated with the array camera designand physical relationship between the calibration target and the arraycamera. The determined parallax shift is accounted for by systematicshifts of the translation data determined as described above inaccordance with many embodiments.

Certain optical characteristics of the individual imaging components,principally those items known as camera intrinsics including but notlimited to focal length, radial distortions, astigmatism and othergeometric distortions, may lead to increased alignment error or adistorted final image. If known, these characteristics may be used tofurther modify the translation data for each imaging element in such amanner that residual alignment errors are reduced and/or the geometricdistortion of the image from the reference imager and, thus, the finaloutput image of the camera are minimized. In a number of embodiments, anappropriate calibration target, for instance, a checkerboard target asdescribed above, is used, along with precise measurement information ofthe calibration set-up, to provide measurements of the relevantdistortion characteristics.

Although specific processes for performing geometric calibration aredescribed above with respect to FIG. 6, any of a variety of processescan be utilized to determine scene independent geometric correctionsthat can be applied to image data captured by an array camera inaccordance with embodiments of the invention. Processes for performingMTF calibration in accordance with embodiments of the invention arediscussed further below.

MTF Calibration

The individual imaging components in an array camera often exhibitnon-uniformities due to the limits of the current manufacturingtechnologies and associated manufacturing processes; and variation inthe manufacturing materials. Some of these non-uniformities exhibitthemselves as variations in the resolving power of the optics over theField of View (FoV). The generation of high resolution images from a setof low resolution imaging devices in an array camera can rely upon anaccurate understanding of, adaptation to, and/or correction of thesevariations. As such, a MTF calibration process may be performed tocharacterize the “sharpness” of the optics over the FoV of each of theimaging components in the array camera. The “sharpness” information maythen be used to adjust the image data received from each imagingcomponent to provide data from the various imaging components thatessentially has a uniform MTF. A process for performing MTF calibrationin accordance with embodiments of this invention is show in FIG. 7.

Process 700 includes capturing images of a target at one or moredistances with the array camera (705). The capturing of the imagesgenerates image data of the target from each of the imaging componentsfor each of the images captured. The target should include one or moreslanted edge structures. For purposes of this discussion, a slanted edgestructure is a structure with a relatively light area and a relativelydark area with a boundary that is a straight line that may beapproximately 6 degrees from vertical. Typically, the distances used tocapture the images will range from approximately Hyperfocal (HF)/2 toHF*2. For example, in an array camera having an HF of 40 cm, 10 or moreimages may be captured between the distances of 10 cm to 2 m. In otherembodiments, any of a variety of distances to targets can be utilized asappropriate to the requirements of a specific application.

After the image data is captured, the following is performed for eachimaging component (730). The image data of the captured images from aselected imaging component is retrieved (710). The image data for eachparticular image of the selected imaging component is analyzed togenerate MTF data over FoV or Field Height (FH) (715). The analysis maybe completed by measuring the MTF at various FoV positions in the imagedata. The MTF is then plotted as a function of distance for each of theFoV positions. The position that yields a peak value for each FoVposition estimates the best focus distance for the selected imagingcomponent and can be included in the MTF information for the imagingcomponent.

A data structure that stores the MTF information for the selectedimaging component gathered from the analysis is then generated (720) andstored (725) for use in the image processing pipeline. In otherembodiments, any of a variety of data structures and/or techniques canbe utilized to store MTF information obtained during calibration asappropriate to the requirements of specific applications.

While the above description contains many specific embodiments of theinvention, these should not be construed as limitations on the scope ofthe invention, but rather as an example of embodiments thereof.Accordingly, the scope of the invention should be determined not by theembodiments illustrated, but by the appended claims and theirequivalents.

What is claimed:
 1. A method for calibrating an array camera using aprocessing system wherein the array camera includes a plurality ofimaging components where the plurality of imaging components includes areference imaging component and a plurality of associate imagingcomponents associated with the reference imaging component, the methodcomprising: performing a defect detection process that includes:identifying pixel defects in each of the plurality imaging components,and determining whether pixel defects in a particular imaging componentmay be corrected in image processing using image data from at least oneother one of the plurality of imaging components; and performing ageometric calibration process that includes determining translationinformation that relates image data from particular pixels in areference imaging component to image data from corresponding pixels ineach associate imaging component associated with the reference imagingcomponent.
 2. The method of claim 1 further comprising performing aModulation Transfer Function (MTF) calibration process on each of theplurality of imaging components in the array camera that includesdetermining MTF data over the Field of View (FoV) or the Field Height(FH) for each of the imaging components.
 3. The method of claim 2wherein the MTF calibration process further comprises: capturing imagedata of a target with each of the plurality imaging components;analyzing the image data captured by the plurality of imaging componentsto determine MTF data over one of a FoV and FH for each of the imagingcomponents wherein the MTF data for each of the imaging components isgenerated from the MTF detected in the image data; and storing the MTFdata for each of the imaging components.
 4. The method of claim 1further comprising performing a photometric calibration process of eachof the plurality of imaging components in the array camera that includesdetermining image data adjustments for each of the imaging componentsthat provide image data from each of the imaging components with auniform intensity profile.
 5. The method of claim 4 wherein theperforming the photometric calibration comprises: capturing image dataof a flat field target under controlled lighting conditions with each ofthe plurality of imaging components in the array camera; analyzing theimage data of each of the imaging components to determine a per pixeladjustment needed to match coŝ4 fall-off response for an image; andgenerating a grid of adjustments for each of the plurality of imagingcomponents wherein each adjustment in the grid indicates the adjustmentinformation for data from a set of pixels in the particular imagingcomponent needed to match the coŝ4 fall-off response for an image. 6.The method of claim 1 wherein the identifying of defects in each of theimaging components comprises: capturing a first set of image data of afield target under high intensity lighting with each of the plurality ofimaging components; capturing a second set of image data of a fieldtarget under low intensity lighting; analyzing the first and second setsof image data to identify each pixel having an unexpected response valuein each imaging component; flagging the identified pixels with acategory indicating the category unexpected response received for eachimaging component; and storing the identified pixels with the flaggedcategories.
 7. The method of claim 6 wherein the detecting of defects ineach imaging component further comprises: determining the number of eachcategory of defects identified in each imaging component; comparing thenumber of each category of defects to a threshold; and generating adisposition output for a particular imaging component in response to thenumber of defects in a category for the particular imaging componentbeing greater than the threshold.
 8. The method of claim 6 wherein thedetermining of whether a defect may be corrected during image processingcomprises: selecting a defect in a first one of the plurality of imagingcomponents in the array camera and determining a parallax translationpath of the defect over a range of supported distances; determining aparallax translation path over the range of supported distances fromdefects in other ones of the plurality of imaging components of thearray camera; determining whether a predetermined number of parallaxtranslation paths of defects in other ones of the plurality of imagingcomponents intersect the parallax translation path of the selecteddefect; and generating a disposition output for the first one of theimaging components having the selected defect if a predetermined numberof parallax translation paths for defects from other images componentsintersect the parallax translation path of the selected defect.
 9. Adevice for calibrating an array camera including a plurality of imagingcomponents where the plurality of imaging components include a referenceimaging component and a plurality of associate imaging componentsassociated with the reference imaging component comprising: a memory;and a processor configured via one or more applications stored in thememory to: perform a defect detection process that includes: identifypixel defects in each of the plurality imaging components, and determinewhether pixel defects in a particular imaging component may be correctedin image processing; and perform a geometric calibration process thatincludes determining translation information that relates image datafrom particular pixels in a reference imaging component to image datafrom corresponding pixels in each associate imaging component associatedwith the reference imaging component.
 10. A machine readable mediumcontaining processor instructions, where execution of the instructionsby a processor causes the processor to perform a process for calibratingan array camera including a plurality of imaging components where theplurality of imaging components include a reference imaging componentand a plurality of associate imaging components associated with thereference imaging component, the process comprising: performing a defectdetection process that includes: identifying pixel defects in each ofthe plurality imaging components, and determining whether pixel defectsin a particular imaging component may be corrected in image processing;and performing a geometric calibration process that includes determiningtranslation information that relates image data from particular pixelsin a reference imaging component to image data from corresponding pixelsin each associate imaging component associated with the referenceimaging component.